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1.0  INTRODUCTION 

This  report  presents  the  results  of  MITRE  Project  86080.  The  objective 
of  this  task  was  to  develop  and  evaluate  preliminary  design  concepts  for 
modeling  command  and  control  (C^  on  the  hypercube  parallel  processing 
computer  architecture  using  the  associated  Time  Warp  operating  system. 
MITRE  performed  this  task  in  support  of  the  Army  Model  Improvement 
Program  (AMIP)  Management  Office  (AMMO)  systems  research  and  planning 
efforts  required  as  part  of  the  development  of  a  new  family  of  Army  models. 

Command  and  control  can  be  thought  of  as  a  large  complex  system  of 

-ft, 

facilities,  equipment,  communications,  procedures,  and  personnel ^  which 
provide  the  means  through  which  the  Secretary  of  the  Army,  Navy,  or  Air 
Force;  the  Chiefs  of  Staff;  and  Commanders  in  the  chajn-of-command 

^  ■{  2k<?,'C  '/■  ;/ 

exercise^command  and  control  of  forces  and  resources,,  in  performing  the 
missions  and  functions  assigned  to  them.  Modeling  CJ  ^  provides  a  means  of 
analyzing  the  process  and  the  effects  of  alternative  doctrine,  tactics,  and  C?^ 
systems.  The  size  and  complexity  of  the  command  and  control  decision 
process  make  it  difficult  to  model;  simulation  is  one  means  of  making  the 
modeling  problem  tractable.  • — v  /  x 

Certain  types  of  simulations  provide  a  computational  means  of  exploring 
complex,  irregular,  discrete  systems  that  are  difficult  to  analyze  by  other 
methods.  Although  simulation  is  a  powerful  tool,  computational  restrictions 
often  mean  that  a  simulation  can  take  days  of  computer  time  to  simulate  an 
hour  of  real  time.  For  example,  one  C^  simulation  requires  two  to  six  hours 
of  computer  time  to  simulate  one  hour  of  real  time.  Not  only  a  decrease  in 
the  ratio  of  simulation  time  to  real-world  time,  but  also  a  more  accurate 
reflection  of  the  real  world  is  desirable  for  applications  to  C^  problems. 
However,  speed  and  accuracy  are  typically  conflicting  objectives;  for 
example,  an  increase  in  information  may  result  in  both  an  increase  in 
accuracy  and  a  decrease  in  simulation  speed. 

More  faithful  and  faster  simulations  are  required  for  the  efficient 
and  effective  evaluation  of  emerging  systems  as  well  as  doctrine.  Object- 
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oriented  programming  ^  is  one  technique  that  has  proved  useful  in  building 
better  C  simulations  and,  hence,  achieving  better  C  representations.  (For 
further  information  on  object-oriented  programming,  see  Section  2.1.) 
However,  advances  in  large-scale  simulations  such  6is  object-oriented 
simulations  (and  symbolic  programs,  in  general)  have  been  constrained  by  the 
serial  nature  of  current  traditional  computer  architectures.  Single-processor 
architectures  limit  a  simulation  because  each  simulation  event  must  take  its 
turn  executing  on  the  processor.  The  advent  of  parallel  computer 
architectures  has  opened  up  a  whole  new  area  of  investigation  into  improving 
the  execution  time  of  these  simulations. 

A 

Existing  C  models  have  to  work  around  trying  to  simulate  parallel, 
independent  real-world  events.  The  use  of  parallel  computing4has  the 
advantage  that  it  would  no  longer  require  the  transformation  of  parallel  event 
representations  into  a  sequential  time  sequence.  The  hope  is  that  distributing 
discrete  portions  of  a  single  simulation  among  many  processors  in  parallel  will 
speed  up  execution  time  by  magnitudes  of  10  to  100. 

1.1  Objective 

The  objective  of  this  effort  was  to  assist— AMMO  by^developw^and 
evaluating^  design  concepts  for  modeling  command  and  control  on  the 
hypercube  parallel  processing  computer  architecture  using  the  associated 
Time  Warp  operating  system.  In  particular,  the  evaluation  was  to  be 
responsive  to  two  basic  questions; 


'o  Can  Time  Warp  on  a  hypercube  architecture  be  used  in  conjunction 
with  object-oriented  techniques  to  significantly  speed  up  the 
processing  time  associated  with  command  and  control  modeling? 

^o  What  are  the  resource  implications  (memory,  communications, 
input/output)  for  the  use  of  a  Time  Warp  and  hypercube 
mechanism? 

f] 

1.2  Approach 

The  key  to  achieving  an  efficient  implementation  of  a  simulation  on  a 
hypercube  is  the  careful  mapping  of  the  simulation  to  the  processors.  A  poor 
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placement  of  a  simulation  on  the  hypercube  could  result  in  worse  than 
serial  execution  time;  on  the  other  hand,  an  optimal  placement  strategy  is  not 
obvious.  Our  approach  to  finding  this  optimal  strategy  can  be  described  by 
three  related  tasks.  The  first  task  finding  was  to  evaluate  whether  Time  Warp 
and  hypercube  could  be  integrated  with  an  object-oriented  simulation,  a 
strategy  which  assigned  portions  of  the  simulation  to  the  different  processors 
to  take  advantage  of  the  parallel  processing.  The  second  task  was  to  develop 
a  means  of  evaluating  alternative  assignment  strategies  so  that  one  strategy 
can  be  judged  better  than  another  strategy.  The  final  task  was  to  develop  an 
assignment  optimization  strategy  to  take  advantage  of  the  gains  of  parallel 
processing. 


1.3  Organization 

The  remainder  of  this  report  is  organized  as  follows.  Section  2.0 
contains  background  material  on  object-oriented  programming,  hypercube,  and 
Time  Warp.  Section  3.0  contains  the  design  considerations  for  the  placement 
of  an  object-oriented  program  on  a  hypercube.  Section  4.0  details  the  design 
decisions  involved  with  the  assignment  of  the  simulation  to  hypercube 
processors  and  an  evaluation  strategy  which  can  be  used  to  evaluate 
alternative  assignment  strategies.  Section  5.0  presents  our  findings  and 
observations,  while  the  study’s  conclusions  are  presented  in  Section  6.0.  The 
Appendix  contains  a  briefing  presented  to  AMMO  outlining  both  the  project 
and  MITRE'S  conclusions. 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


2.0  BACKGROUND 


2 

The  limited  representation  of  C  in  existing  combat  simulations  is  one 
of  several  concerns  that  prompted  the  establishment  of  AMMO,  and  it 
continues  to  be  a  concern  of  the  AMMO  office.  AMMO's  charter  is  to  develop 
a  comprehensive  and  well-integrated  Army  Modeling  capability  at  the  earliest 
date.  MITRE  previously  has  supported  AMMO  in  its  development  of  the 
functional  area  representative  objectives  for  the  Corps  and  Division 
Evaluation  Model  (CORDIVEM).  These  were  developed  utilizing  work 
previously  done  on  the  functional  segment  subsystem  specifications.  At  the 
same  time,  MITRE  conducted  a  technical  assessment  of  selected  Army 

O 

simulation  models  as  candidates  for  the  automated  CORDIVEM  °.  Still 
another  task  conducted  in  support  of  the  AMMO  office  was  the  evaluation  of 

Q 

object-oriented  programming  for  use  in  Army  simulation  modeling  . 

As  part  of  this  last  evaluation,  MITRE  used  the  RAND  Corporation’s 
Rule  Oriented  Simulation  System  to  develop  a  ground  combat  simulation 
entitled  the  Battlefield  Environment  Model  (BEM)  to  examine  the  application 
of  object-oriented  programming.  This  MITRE  evaluation  showed  that  the 
benefits  associated  with  the  object-oriented  approach  to  simulations  include 
increased  modifiability  and  intelligibility  and  improved  performance 
characteristics;  however,  this  was  at  a  cost  of  large  computer  memory  and 
slow  running  time.  Parallel  processing  is  regarded  as  a  possible  means  of 
getting  acceptable  running  times  while  keeping  the  increased  representation. 

Use  of  the  hypercube  architecture  will  require  the  placing  of  an  object- 
oriented  simulation  on  the  multiprocessors;  this  requires  the  breaking  of  the 
simulation  into  manageable  pieces,  the  assignment  of  these  pieces  to 
processors,  and  the  synchronization  of  the  various  simulation  pieces  as  they 
will  be  running  at  different  simulation  speeds.  This  is  the  problem  that  the 
Time  Warp  operating  system  is  designed  to  handle,  the  time  synchronization 
problem.  The  following  sections  will  discuss  object-oriented  programming, 
hypercube  and  Time  Warp. 
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Object-oriented  simulations  are  centered  around  objects  (actors  in  the 
simulation)  which  can  communicate  cmly  by  passing  messages  between  objects 
and  are  distinguished  by  having  an  action  or  a  behavior  in  the  simulation 
initiated  by  the  receipt  of  a  message  sent  by  another  object.  Objects  in  this 
type  of  simulation  have  properties  and  behaviors  which  represent  the 
characteristics,  activities,  and  capabilities  of  the  objects  being  simulated. 
The  receipt  of  messages  triggers  behaviors  in  these  objects  akin  to  real-world 
objects  receiving  communications  and  causes  the  objects  to  perform  certain 
actions.  The  objects  are  created  in  a  hierarchy  which  permits  inheritance  of 
properties  and  behaviors;  e.g.,  a  generic  Army  battalion  can  be  created  with 
properties  such  as  speed  and  number  of  trucks.  This  generic  battalion  then 
can  be  replicated  with  all  its  properties  and  each  replication  can  inherit  these 
properties  from  the  generic  battalion.  In  summary,  the  primary 
characteristics  of  object-oriented  programming  are  the  use  of  objects, 
message  passing  characteristics,  and  inheritance  of  properties  and  behaviors. 

2.1.1  Objects 

There  are  two  types  of  objects  used  in  object-oriented  simulations: 
basic  and  auxiliary  objects.  Basic  objects  are  the  actors  in  the  simulation  in 
that  they  have  some  real-world  representation  such  as  the  military  units  being 
simulated.  There  can  be  a  one-to-one  correspondence  between  real  military 
objects  and  basic  simulation  objects,  e.g.,  an  Army  battalion  and  simulation 
object  representing  a  battalion,  or  possibly  military  objects  can  be 
represented  by  more  than  one  type  of  simulation  object.  The  basic  simulation 
objects  must  have  characteristics  and  behaviors  sufficiently  defined  '‘or  them 
to  portray  adequately  the  system  or  units  being  represented.  All  action  in  the 
simulation  is  controlled  through  the  passing  of  messages.  Simulation  object 
behaviors  are  in  the  form  of  IF-THEN  rules  which  list  the  actions  to  be  taken 
by  an  object  on  receipt  of  particular  messages. 

The  second  type  of  simulation  object,  an  auxiliary  object,  has  no  real- 
world  counterpart,  but  is  used  for  support  and  control  of  the  simulation. 


Auxiliary  objects  can  be  used  to  support  the  simulation  in  various  ways  but  are 
not  directly  involved  in  the  simulation.  Auxiliary  objects  can  assist  in  the 
construction  and  conduct  of  the  simulation;  for  example,  an  auxiliary  object 
might  be  used  to  centralize  most  of  the  mathematical  computation,  thus 
making  the  basic  object  behaviors  cleaner  and  therefore  more  readable. 
Auxiliary  objects  are  also  used  to  centralize  common  procedures  such  as 
movement  calculations  and  can  be  used  to  maintain  databases,  such  as  a  time- 
dependent  terrain  database  to  which  simulation  objects  need  access.  For 
example,  an  actor  called  Terrain  would  be  used  because  the  overhead  involved 
with  keeping  the  entire  terrain  database  on  each  object's  property  list  is 
undesirable. 

2.1.2  Message  Passing 

Simulation  objects  pass  messages  between  themselves  which  activate 
object  behaviors.  There  are  two  types  of  messages  in  the  simulation;  those 
which  simulate  real-world  communications  and  those  which  might  be  called 
system  messages.  This  first  type  of  message  triggers  actor  behaviors  which 
simulate  real  military  unit  reactions  to  messages.  It  is  this  combination  of 
message  passing  of  the  first  type  and  rule-based  behavior  which  provide  a 
close  parallel  to  the  communications  and  the  command  and  control  which  are 
necessary  for  a  credible  combat  simulation.  The  latter  message  type  includes 
those  messages  which  might  be  used  for  simulation  control  or  for  data  access. 

2.1.3  Simulation  Time 


Because  all  action  in  the  simulation  takes  place  through  behaviors 
activated  by  simulation  messages,  the  object-oriented  simulation  can  easily  be 
executed  in  serial  by  executing  the  simulation  messages  in  time  order.  The 
BEM  is  currently  executed  in  a  time-stepped,  sequential  manner,  e.g.,  all 
actions  scheduled  for  simulation  time  5  are  executed  before  simulation  time 
6.  Sequential  execution  prevents  time  anomalies  which  might  occur  in 
parallel  execution.  Regardless  of  the  type  of  execution,  a  time  history  of  all 
(or  selected)  messages  provides  a  trace  of  cause  and  effect  in  the  simulation. 
We  shall  see  later  that  time  plays  an  especially  important  role  when  the 


object-oriented  simulation  is  transformed  from  its  current  sequential  form 
into  a  form  appropriate  to  parallel  architecture. 


2.1.4  Inheritance  Hierarchy 

In  object-oriented  programming,  specific  instances  of  objects  can  be 
created  from  generic  classes  of  objects,  forming  a  hierarchy  of  objects. 
Descendants  in  a  hierarchy  can  inherit  the  properties  and  behaviors  of 
hierarchical  ancestors;  just  as  a  child  inherits  characteristics  from  both 
parents.  For  another  example,  a  generic  simulation  unit,  "Tank  Co,"  could 
inherit  properties  and  behaviors  from  a  higher  simulation  unit  called  "Action 
Unit"  (Figure  1).  The  advantage  of  this  inheritance  principle  is  that  common 
properties  and  behaviors  need  to  be  entered  only  once  in  the  construction  of 
the  simulation.  This  simplifies  object  modification  and  reduces  the  amount  of 
storage  required. 

2.2  Hypercube 

In  pursuit  of  this  challenge  to  obtain  faster  and  more  accurate 
simulations,  the  U.S.  Army  is  developing  a  prototype  hardware/software 
system  based  on  a  "hypercube"  processor^^.  The  hypercube  is  a 
multiprocessor  architecture  designed  at  California  Institute  of  Technology  and 
built  at  Jet  Propulsion  Lab.  The  primary  advantage  of  a  hypercube  is  that  2*^ 
processors  can  work  in  concert  on  a  particular  problem  and  hopefully  solve  the 
problem  faster  than  one  computer.  This  does  not  mean,  however,  that  if  n 
processors  are  working  on  a  problem,  it  will  be  solved  n  times  faster, 
Furthermore,  it  should  be  noted  that  adding  processors  can  mimic  a 
diminishing  returns  curve;  at  some  point  extra  processors  detract  from  the 
system. 

An  n-dimensional  hypercube  is  constructed  from  2*^  nodes,  each 
containing  an  identical,  independent  processor  and  local  memory,  all 
connected  by  high-speed  links  arranged  in  the  topology  of  an  n-dimensional 
Boolean  hypercube.  Thus  a  3-D  hypercube  (shown  in  Figure  2)  has  8  nodes 
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connected  in  the  pattern  of  an  ordinary  cube,  and  a  4-D  hypercube  has  16 
nodes.  Each  node  has  exactly  n  nearest  neighbors  in  this  topology;  the 
average  distance  between  two  nodes  is  n/2  and  the  maximum  distance  is  n 
"hops”  (where  a  hop  is  the  distance  as  measured  by  processors,  between  any 
two  nodes).  The  current  generation  of  hypercubes  has  no  shared  memory. 
Processors  may  share  information  through  message  passing,  but  there  is  no 
central  memory  or  database. 

Many  different  parallel  processing  architectures  are  under  development, 
each  with  its  own  unique  set  of  characteristics  (e.g.,  shared  memory)  and 
communication  network.  The  hypercube  can  be  considered  a  middle-of-the- 
road  architecture  with  respect  to  two  criteria:  the  average  number  of  hops 
required  for  communication  and  the  number  of  processor  communication 
links.  One  extreme  of  the  multiprocessor  architecture  spectrum  is  where 
each  processor  is  connected  to  two  other  processors,  for  example,  a  ring.  This 
architecture's  disadvantage  is  that  in  order  for  two  processors  to 
communicate,  they  may  have  to  go  through  many  other  processors  and  incur 
large  communications  overhead.  An  example  of  the  other  end  of  the 
architecture  spectrum  is  where  each  processor  is  connected  to  every  other 
processor.  The  number  of  processors  allowed  in  most  architectures  is  limited 
by  the  number  of  input/output  ports  on  each  processor,  typically  15  for  the 
hypercube. 


2.3  Time  Warp 

In  conjunction  with  the  development  of  hypercube,  the  Army  is  also 
supporting  the  development  of  the  Time  Warp  operating  system,  a  specialized 
operating  system  for  distributed  simulations.  Time  Warp  is  a  mechanism 
invented  at  the  RAND  Corporation  in  1981  which  allows  the  objects  to 
proceed  asynchronously  through  simulation  time;  this  means  that  some  of  the 
objects  are  allowed  to  progress  ahead  in  simulation  time  while  others  lag 
behind.  Time  Warp  deals  with  the  time  anomalies  caused  by  the  asynchronous 
processing. 


The  Time  Warp  mechanism  allows  an  object  to  go  forward  in  simulation 
time  until  it  receives  an  event  message  that  it  should  have  executed  in  its 
simulated  past.  An  object  can  receive  messages  in  its  past  because  each 
processor  can  be  at  a  different  simulation  time.  For  example,  suppose  a 
military  battalion  object's  simulation  time  is  0235  hours  while  the  division 
headquarters'  simulation  time  is  0130,  and  that  the  division  headquarters 
directs  the  battalion  to  move  now.  Division  headquarters  "now"  is  0130;  this 
time  is  in  the  battalion's  past.  The  battalion  must  therefore  rollback  to  a 
simulation  time  earlier  than  0130  and  start  re-executing  its  part  of  the 
simulation  (Figure  3);  the  object  is  able  to  roll  back  to  an  earlier  time  because 
it  has  saved  previous  messages  and  states.  The  insertion  of  the  new  message 
may  change  an  object's  future,  and  as  a  result  its  old  set  of  future  states  may 
no  longer  be  valid.  However,  this  implies  that  some  of  the  messages  the 
object  has  sent  out  may  also  be  invalid.  Therefore,  as  the  object  goes  forward 
in  simulation  time,  it  must  check  to  see  if  it  has  sent  out  any  inappropriate 
messages  or  caused  any  inappropriate  side  effects.  To  correct  this  problem, 
as  the  object  goes  forward  in  simulation  time,  it  checks  each  message's 
validity.  If  the  object  does  find  an  invalid  message,  it  sends  an  anti-message 
to  annihilate  it;  any  messages  which  remain  valid  are  left  unchanged.  The 
receipt  of  an  anti-message  can  cause  a  secondtu’y  rollback.  The  Time  Warp 
however  does  guarantee  a  forward  progression  in  time  and  that  the  process  is 
free  of  deadlock The  main  feature  distinguishing  it  from  other  distributed 
simulation  mechanisms  (e.g.,  those  for  scientific  applications)  is  its  reliance 
on  distributed  rollback  and  "anti-messages"  for  synchronization  and  flow 
control^ 
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3.0  PORTING  CONSIDERATIONS 

A  C  object-oriented  simulation  consists  of  a  collection  of  simulation 
objects  representing  military  units  with  real  behaviors,  procedures,  and  Iswge 
common  databases.  The  simulation  objects  are  used  to  mimic  the  actions  of 
military  units,  e.g.,  communicating  messages,  processing  information, 
changing  location,  etc.  Simulation  objects  are  used  to  represent  real  military 
units  performing  (sequential)  processes  that  can  executed  in  parallel  and  can 
communicate  with  each  other  by  exchanging  messages.  In  Time  Warp,  these 
messages  are  time  stamped  with  a  simulation  time,  and  the  order  of  event 
execution  is  determined  by  the  time-stamp  on  a  message. 

Real  events  are  events  which  occur  in  parallel  in  the  physical  world  and 
are  independent  of  one  another  and  can  be  simulated  concurrently.  This  does 
not  mean  that  a  simulation  with  100  objects  on  100  processors  can  be  made  to 
go  100  times  faster  through  the  use  of  parallel  architectures;  it  does  mean 
that  the  degree  of  achievable  computational  concurrency  is  directly 
proportional  to  the  degree  of  physical  concurrency  or  independence  in  a 
model  This  means  that  if  the  real  events  are  both  sequential  and  dependent, 
it  is  very  difficult  to  separate  events  and  execute  them  in  parallel.  This  level 
of  potential  computational  concurrency  provides  an  upper  bound  to  how  much 
a  simulation  can  potentially  be  speeded  up;  if  there  is  little  concurrency  in  a 
particular  representation,  its  implementation  on  a  parallel  architecture  will 
not  greatly  impact  its  execution  time. 

The  porting  of  an  object-oriented  simulation  onto  a  hypercube 
architecture  requires  that  the  simulation  be  placed  on  more  than  one 
processor.  This  means  that  the  representation  must  be  decomposed  in  such  a 
way  that  its  subsets  can  be  assigned  to  different  processors,  more  than  one 
subset  can  be  assigned  to  one  processor.  Note  that  a  poor  decomposition  of 
the  simulation  onto  the  parallel  processor  can  result  in  a  worse  than  serial 
implementation. 

Two  strategies  are  needed  to  port  an  object-oriented,  message-passing 
simulation  onto  a  Time  Warp/hypercube  system;  the  assignment  of  objects  to 
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processors  and  the  assignment  of  information  and  procedures  to  objects. 
These  assignment  strategies  are  discussed  in  Sections  3.2  and  3.3. 

3.1  Constraints 


There  are,  of  course,  factors  that  impose  constraints  on  any  assignment 
strategy:  the  size  of  the  processor  memory,  the  processor  speed,  and  the 
speed  of  communications.  Memory  size  limits  the  number  of  objects  on  a 
processor  and  similarly  limits  the  amount  of  information  and  procedures  on 
each  processor.  The  optimal  number  of  processors  for  a  particular 
implementation  will  also  be  a  function  of  memory  size;  the  smaller  the 
memory,  the  greater  the  number  of  processors  required.  Processor  speed  will 
constrain  the  number  of  objects  which  can  be  assigned  to  a  processor;  if  a 
slow  processor  is  assigned  many  active  units,  it  will  proceed  very  slowly 
through  simulation  time.  Slow  machine  communications  will  strongly 
influence  the  assignment  of  information  to  objects  and  the  assignment  of 
objects  to  processors;  slow  machine  communication  speed  makes  it  cheaper  to 
store  information  than  to  access  it  elsewhere.  Furthermore,  slow  machine 
communication  speeds  will  make  the  simulation  slower.  The  optimization 
strategies  associated  with  these  constraints  sure  discussed  in  Section  4.0. 

3.2  Object  Assignment  Strategies 

There  are  three  basic  strategies  for  assigning  objects  to  processors.  The 
first  strategy  is  to  place  all  objects  on  one  processor;  the  result  is  slow  serial 
implementation.  The  second  strategy  places  each  object  on  its  own  processor. 
A  pure  implementation  of  this  strategy  is  limited  by  the  number  of  processors 
available.  The  third  strategy  clusters  objects  on  processors,  i.e.,  places  more 
than  one  object  on  a  processor.  The  degree  of  this  clustering  is  influenced  by 
processor  characteristics,  simulation  characteristics,  and  the  number  of 
messages  passed  between  objects.  Communications  speed  may  make  it 
desirable  to  place  objects  which  communicate  frequently  within  the  same 
processor,  or  at  least  on  processors  close  to  each  other,  to  reduce 
communications  overhead. 
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3.3  Information  and  Behavior  Assignment  Strategic 


Similarly,  there  are  three  basic  strategies  to  assign  information  and 
behaviors  to  objects.  The  first  strategy  is  to  give  each  object  all  possible 
pertinent  information,  implying  a  high  degree  of  replication  and  associated 
database  synchronization  overhead.  The  second  strategy  is  to  give  each 
object  most  of  its  frequently  used  information.  The  object  then  would  have  to 
query  other  objects  for  information  that  it  does  not  have.  This  means  that 
there  is  a  lower  degree  of  information  replication  throughout  the  system.  The 
advantage  of  this  is  that  synchronization  costs  are  lower;  the  disadvantage  is 
that  an  object  will  have  to  spend  more  time  acquiring  information.  The  last 
strategy  assigns  each  object  a  minimum  of  information.  This  strategy  implies 
very  little  information  replication  and,  therefore,  low  database 
synchronization  costs.  It  does  however  mean  that  each  time  a  unit  needs  a 
piece  of  information  that  it  must  go  acquire  it;  this  strategy  would  have  high 
waiting  time  costs. 

3.4  Expected  Bottlenecks 

There  appear  to  be  two  possible  sources  of  major  bottlenecks  to 
achieving  satisfactory  performance  results  after  problem  decomposition  and 
object  assignment  are  dealt  with.  The  first  of  these  possible  bottlenecks  is 
the  sequential  nature  of  many  command  and  control  processes,  at  least  at  the 
higher  echelons  of  command.  A  high  degree  of  centralized  control  also  would 
make  the  top  level  control  unit  in  the  simulation  a  possible  bottleneck.  The 
second  area  in  which  bottlenecks  may  be  created  is  that  of  large  databases  to 
which  many  objects  need  access  and  which  may  be  time  dependent.  Methods 
for  dealing  with  database  bottlenecks  are  discussed  in  Section  4.4. 


4.0  IMPLEMENTATION  STRATEGY  AND  EVALUATION 


The  performance  of  a  model  on  a  sequential  computer  is  usually 
measured  by  the  number  of  computer  instructions  (and/or  operations)  required 
to  complete  the  simulation.  This  type  of  standard  analysis  is  insufficient  for 
parallel  implementations  because  of  all  the  additional  type  operations  present, 
e.g.,  communication  and  synchronization.  Therefore,  this  standard  measure 
cannot  be  used  to  gauge  the  decisions  which  must  be  made  relative  to  the 
assignment  of  objects  to  processors  and  the  assignment  of  information  and 
procedures  to  each  object. 

Therefore,  an  alternate  evaluation  scheme  is  needed  to  compare 
alternative  assignment  strategies;  one  scheme  is  to  use  total  execution  time 
as  a  global  evaluation  measure.  Execution  time  of  a  simulation  running  on 
multiple  cooperating  processors  can  be  defined  as  the  time  elapsed  from  when 
the  first  processor  begins  until  when  the  last  processor  finishes.  If  the 
execution  time  on  any  given  processor  is  defined  as  the  sum  of  the  busy  and 
idle  times  for  that  processor,  execution  time  will  be  the  same  for  all 
processors.  The  implementation  strategy  should  attempt  to  minimize  the 
execution  time  for  all  processors. 

4.1  Assumptions 

Two  initial  assumptions  are  made  to  facilitate  the  initial  development  of 
the  evaluation  measure;  we  will  discuss  the  ramifications  associated  with  the 
loosening  of  these  assumptions  later  in  the  paper.  The  first  assumption  is  that 
sufficient  processor  memory  is  available.  This  implies  that,  at  least  initially, 
memory  limitations  are  not  an  overriding  concern.  This  is  a  reasonable 
assumption  to  make,  as  the  currently  available  hypercubes  (e.g.,  Intel  and 
Mark  II)  have  a  reasonable  amount  of  memory  (between  .5  and  4  megabytes) 
on  each  processor.  This  assumption  allowed  us  to  hypothesize  what 
information  and  object  assignment  strategies  make  intuitive  sense,  without 
being  bound  by  memory  constraints. 


The  second  assumption  is  merely  that  communication  speed  is 
reasonable  relative  to  this  application.  Slow  communication  speed  would 
strongly  influence  the  assignment  of  information  to  objects  and  objects  to 
processors,  as  it  makes  the  price  of  accessing  information  higher  than  storing 
it.  At  the  very  least,  slow  machine  communication  speed  will  slow  down  the 
simulation  and  increase  the  time  difference  between  sender  and  receiver  of 
messages. 

4.2  Global  Evaluation  Measure 

Execution  time  or  E  is  used  as  the  global  execution  time,  which  can  be 
broken  down  into  a  four-part  sum,  consisting  of  P,  processing  time;  W,  wait  or 
idle  time;  S,  synchronization  time;  and  C,  communication  time,  i.e., 

E=P+W+S+C 

Execution  time  begins  when  the  first  processor  begins  executing  and 
ends  when  the  last  processor  finishes  execution.  Execution  time  is  the  same 
for  all  processors,  however  the  components  of  execution  time  will  vary  by 
processor. 

Processing  time  reflects  all  the  time  spent  by  a  processor  processing 
information  from  one  or  more  objects.  Synchronization  time  consists  of  time 
a  processor  spends  as  part  of  the  synchronization  process  (rollbacks  and  time 
updates).  Wait  time  for  each  processor  consists  of  the  time  spent  idle  as  well 
as  the  time  spent  waiting  for  replies  to  queries.  Communication  time  reflects 
time  the  processor  spends  communicating  values  from  one  processor  to 
another  (transmission  time  only)^. 

The  communication  component  consists  of  three  types  of 
communications:  query,  update,  and  action  messages.  A  query  occurs  when 
an  object  requests  information  from  another  object.  An  object  can  respond  to 
queries  without  rolling  back  because  it  keeps  a  record  of  old  state 
information.  The  amount  of  historical  information  which  can  be  kept  is 
partially  a  function  of  processor  memory  size.  An  update  is  a  change  made  to 
a  database  and  can  cause  rollbacks.  An  action  message  causes  an  object  to  do 
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something  and  can  cause  rollbacks.  Rollbacks  occur  when  either  an  update  or 
an  action  message  arrives  at  a  processor  with  a  time  stamp  earlier  than  the 
processor's  local  simulation  time.  These  components  will  be  referred  to  later 
as  the  query-communication  component,  the  update-communication 
component,  and  the  rollback-communication  component. 

Since  each  processor  within  the  hypercube  is  actually  a  serial  processor, 
the  model  assumes  that  each  processor  is  serial  and  can  perform  only  one 
process  at  a  time.  Otherwise  each  processor  could  do  more  than  one  thing  at 
a  time.  This  would  make  the  processor  a  type  of  parallel  processor,  which 
could  then  be  split  into  several  serial  processors.  The  model  also  assumes  no 
overlap  of  processing,  communication,  and  synchronization  times;  this  implies 
a  processor  can  handle  only  one  task  at  a  time. 

4.3  Assignment  Strategy  Based  on  Object  Characterization 

An  initial  strategy  for  assigning  information  and  procedures  was  chosen 
based  on  the  command  and  control  functions  performed  by  each  object,  and  its 
ease  of  implementation.  Each  military  object  participates  to  some  degree  in 
a  set  of  four  functions:  plan,  coordinate,  direct  and  controL  One  way  of 
modeling  the  C  process  is  to  look  at  the  various  tasks  which  must  be  carried 
out  in  support  of  these  functions.  These  tasks  include  collect  information; 
evaluate  plan,  capabilities,  and  requirements;  develop  decision  and  transmit; 
allocate  forces;  coordinate;  control,  monitor,  and  adjust.  In  order  for  the 
simulation  objects  to  be  able  to  carry  out  these  tasks,  the  simulation  must 
include  time  ordering  of  supporting  subtasks,  data  bases  and  knowledge  bases, 
and  rules  for  processing  information.  For  example,  in  a  military  simulation,  a 
typical  high-level  military  object,  such  as  a  corps  headquarters,  spends  much 
of  its  time  processing  and  fusing  information  in  order  to  make  plans  and  issue 
orders.  To  perform  this  function  it  needs  a  considerable  amount  of  processing 
capabilities  and  a  large  database  which  is  kept  current  by  updates  from  lower 
echelons. 


4.3.1  High-Level  Object  Characterization 


The  characterization  of  an  ctoject's  functions  can  be  used  to  select  an 
initial  information  and  procedure  assignment  strategy  which  appears  to  match 
an  object's  complexity  and  characterization.  This  initial  strategy  implies 
assigning  an  object  all  the  information  appropriate  to  its  characterization; 
e.g.,  if  it  uses  a  great  deal  of  information,  assign  it  large  databases.  For 
example,  at  the  upper  end  of  the  spectrum  in  a  corps-level  simulation  are  the 
corps  and  division  headquarters.  While  these  headquarters  are  few  in  number, 
they  are  complex  in  the  level  of  processing  capability  to  be  represented.  The 
database  maintained  by  top-level  control  units  in  the  simulation  should  be 
both  detailed  and  aggregated  status  information,  and  this  must  be  both  in 
present  and  projected  terms.  The  corps  planning  horizon  is  72  hours,  and  data 
forecasts  are  required  to  support  that  planning.  The  time  horizon  and  larger 
area  of  interest  imply  large  databases  to  contain  the  required  information  for 
planning  and  decisionmaking.  In  keeping  with  the  data  requirements  of 
objects  at  this  level,  initial  strategy  would  be  to  give  a  division  or  corps 
headquarters  simulation  object  all  necessary  information  and  procedures 
because  they  need  a  large  amount  of  information  and  processing  capability  to 
fulfill  their  mission. 

Given  an  initial  information  assignment  strategy  and  characterization  of 
an  object,  the  hypothesized  performance  components  could  be  analyzed  for 
the  implications  this  initial  strategy  has  for  the  execution  measure.  For 
example,  the  processing  component  in  this  case  would  probably  be  high 
because  this  type  of  object  primarily  plans  and  processes  information.  The 
component  associated  with  query  communication  would  probably  be  moderate 
due  to  queries  from  low-level  objects.  The  action-communication  component 
is  a  function  of  the  particular  simulation  and  independent  of  the  amount  of 
stored  information.  The  large  amount  of  information  associated  with  this 
strategy  implies  a  high  degree  of  information  replication;  therefore,  the 
update  communication  overhead  probably  would  be  high. 


The  amount  of  rollback  due  to  action  messages  is  a  function  of  the 
virtual  time  difference  between  the  sender  and  the  receiver  of  the  action 
message.  The  frequency  of  action  rollbacks  is  independent  of  the  amount  of 
stored  information.  No  rollback  is  caused  if  the  update  is  for  the  current  or  a 
future  time.  The  number  of  rollbacks  caused  by  updates  may  in  some  part  be 
a  function  of  the  number  of  updates,  but  more  importantly  it  is  a  function  of 
the  virtual  time  difference  between  sender  and  receiver  of  updates. 

4.3.2  Optimization  Strategy  for  High-Level  Objects 

An  appropriate  question  is  whether  this  initial,  intuitive  strategy  can  be 
improved  relative  to  the  execution  time  measure?  In  fact,  the  strategy 
probably  can  be  improved;  however,  the  optimization  is  not  straightforward 
because  of  the  complex  inter-relationships  of  the  execution  measure 
components.  High  amounts  of  processing  time  for  a  particular  node  could 
mean  that  the  local  virtual  time  of  that  particular  processor  might  lag  behind 
that  of  other  processors  and  thus  cause  frequent  rollbacks.  One  obvious  way 
of  reducing  this  component  would  be  to  place  the  object  onto  more  than  one 
processor,  a  type  of  load  balancing.  This  can  be  done  by  dividing  the  functions 
of  the  represented  object  and  creating  two  or  more  simulation  objects. 

The  high  update  component  also  could  be  reduced  if  the  amount  of 
stored  information  were  decreased.  However,  this  decrease  would  mean  that 
object  would  now  have  to  query  other  objects  for  information  and  thus  spend 
more  of  its  time  communicating  queries  and  waiting  for  responses. 

Rollbacks  can  be  kept  low  by  minimizing  the  virtual  time  difference 
between  sender  and  receiver  of  updates/actions.  The  virtual  time  difference 
is  minimized  when  all  the  processors  run  at  approximately  the  same 
simulation  speed.  This  can  be  accomplished  by  partitioning  objects  and  by 
placing  the  partitions  on  different  processors. 

4.3.3  Characterization  of  a  Low-Level  Object 


A  resolution  object  (i.e.,  the  lowest  level  object  being  simulated)  such  as 
an  infantry  battalion  is  concerned  with  its  present,  or  near-term,  status  as 
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well  as  the  mission  and  plan  which  have  been  given  it.  It  is  also  concerned 
with  and  requires  information  regarding  its  status  and  that  of  opposing 
forces.  It  spends  most  of  its  time  executing  orders,  obtaining  information 
from  other  objects,  and  sending  information  to  higher  echelons.  The  object's 
characteristics  indicate  the  amount  of  time  spent  planning  and  processing 
information  is  relatively  smalL  The  characterization  of  an  object  can  again 
be  used  to  select  an  initial  information  and  procedure  assignment  strategy 
which  appears  to  match  a  resolution  object's  characterization,  e.g.,  low-level 
military  objects  such  as  an  infantry  battalion  receive  minimal  information  and 
procedures  because  the  processing  of  information  is  not  a  primary  function. 

Given  this  initial  assignment  strategy  and  the  characterization  of  the 
object,  one  can  hypothesize  about  the  components  of  the  evaluation 
measure.  The  processing  component  would  be  low  because  this  type  of  object 
does  little  processing  of  information  (because  there  is  little  synthesis  of 
information  from  subordinate  units)  and  its  special  needs  can  be  filled  by 
database  or  specialized  objects  (procedure  objects  which  provide  functions 
such  as  Mathematician).  The  query-communication  component  would  be  high 
because  the  object's  basic  function  is  to  obtain  information,  such  as  enemy 
position,  from  other  objects.  The  action-communication  component  is  a 
function  of  the  simulation  only  an  independent  of  the  information  assignment 
strategy.  The  minimal  database  implies  minimal  information  replication  and 
update  overhead.  Rollbacks  are  a  function  of  the  time  differential  between 
sender  and  receiver  of  a  communication;  the  farther  behind  the  sender  is  in 
virtual  time,  the  greater  the  magnitude  of  the  rollback. 

4.3.4  Optimization  Strategy  for  Low-Level  Objects 

Again,  an  appropriate  question  is  whether  this  initial,  intuitive 
assignment  strategy  can  be  improved  relative  to  the  execution  time 
measure?  Similarly,  the  answer  is  in  the  affirmative.  Low  amounts  of 
processing  imply  that  an  object  can  race  ahead  relative  to  simulation  time; 
however,  this  may  not  always  be  a  desirable  situation.  If  a  battalion  object 
were  far  ahead  of  its  division  in  local  virtual  time,  all  the  orders  issued  by  the 
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division  would  cause  rollbacks  in  the  battalion,  thereby  increasing 
synchronization  overhead.  To  prevent  frequent  rollbacks,  it  may  be  desirable 
(although  nonintuitive)  to  slow  the  object  (processor)  down  by  clustering 
several  low-level  objects  on  the  processor. 

The  high  query  and  wait  time  components  can  be  reduced  through 
judicious  additions  to  the  object's  information  and  procedures;  however,  any 
increase  in  the  size  of  the  stored  database  will  also  increase  the  update- 
communication  component.  The  rollback  component  can  be  kept  low  by 
minimizing  the  local  virtual  time  difference  between  the  sender  £uid  receiver 
of  updates  or  actions.  In  order  to  keep  all  the  processors  running  at 
approximately  the  same  simulation  speed,  the  low-level  objects  must  be 
prevented  from  getting  too  far  ahead  of  some  of  the  larger  objects;  this  can 
be  accomplished  by  clustering  several  of  the  low-level  objects  on  a  processor. 

4.4  Common  Databases  and  Procedures 
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C  simulations  have  two  other  components  which  must  be  dealt  with  to 
take  advantage  of  parallel  processing:  procedures  and  common  databases. 
Procedures  are  collections  of  related  function  routines,  e.g.,  movement  or 
mathematical  functions.  The  procedure  calls  are  grouped  together  to  simplify 
the  behaviors  of  the  military  objects;  however,  it  is  feasible  to  decompose 
these  objects,  replicate  the  functions,  and  place  copies  of  the  replications 
with  each  object.  Common  databases  are  collections  of  related  information. 
Both  of  these  components  can  be  handled  through  the  use  of  auxiliary 
objects.  However,  auxiliary  objects  frequently  become  simulation  bottlenecks 
because  of  the  number  of  simulation  objects  trying  to  use  functions  or  access 
databases  of  the  auxiliary  objects.  Auxiliary  objects  should  be  avoided,  if 
possible,  or  implemented  very  carefully  to  minimize  the  bottleneck.  The 
problem  becomes  somewhat  akin  to  the  distributed  database  problem:  how 
many  times  should  a  database  be  replicated  so  that  bottlenecks  are  prevented 
and  at  the  same  time  high  communications  overheads  are  not  incurred. 

Database  objects  pose  a  special  problem  since  they  are  frequently  time 
dependent  and  the  frequency  of  change  for  the  common  databases  varies 
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considerably;  e.g.,  ground  truth  changes  frequently  while  terrain  changes 
infrequently.  Databases  with  high  frequency  of  change  cannot  be  freely 
replicated,  because  it  can  be  difficult  to  keep  the  replications  consistent.  The 
frequency  of  change  also  impacts  the  frequency  of  rollbacks.  Objects  usually 
do  not  possess  the  discrimination  to  determine  whether  a  rollback  affects 
them.  If  a  change  in  a  pertinent  database  is  made,  a  rollback  must  be 
initiated. 

The  database  optimization  strategy  is  primarily  a  function  of  the 
frequency  of  the  database  changes.  For  databases  with  low  frequency  of 
change,  some  form  of  replication  on  several  processors  can  prevent  simulation 
bottlenecks.  The  form  of  the  replication  varies.  For  those  databases  with  low 
frequency  of  change,  it  is  possible  to  copy  the  databases  and  to  assign  copies 
of  the  database  to  the  objects  as  needed.  For  databases  with  high  frequency 
of  change,  the  overhead  associated  with  keeping  many  copies  is  prohibitive. 
An  alternative  strategy  would  be  to  judiciously  partition  the  database  and  to 
place  the  partitions  on  different  processors;  this  would  decrease  the  wait 
required  to  access  the  database.  Both  strategies  should  attempt  to  minimize 
the  local  virtual  time  difference  between  the  database  and  the  objects  which 
use  it  in  order  to  minimize  the  number  of  rollbacks. 

4.5  Tightening  of  Assumptions 

If  the  assumptions  made  earlier  do  not  in  fact  hold,  one  might  use  a 
different  strategy  and  obtain  different  results.  For  example,  if  the  processor 
memory  is  limited,  it  is  not  feasible  to  give  each  object  its  own  large 
database.  Similarly,  a  tight  memory  constraint  limits  the  number  of  old  state 
copies  that  could  be  saved,  the  degree  of  object  clustering  possible,  and, 
therefore,  the  ease  of  rollback,  the  degree  of  load  balancing  possible,  and  the 
amount  of  information  and  procedures  which  could  be  stored.  Furthermore,  if 
the  processor  memory  is  restrictive  and  only  a  limited  number  of  processors 
are  available,  the  possible  gains  for  parallelizing  a  large  simulation  are  small. 

If  communication  speed  is  a  tight  constraint,  it  more  strongly  influences 
the  assignment  of  information  to  objects,  and  objects  to  processors,  as  it 
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makes  the  price  of  accessing  information  higher  than  storing  it.  Additionally, 
slow  machine  communication  speed  slows  down  the  simulation  and  increases 
the  time  difference  between  sender  and  receiver  of  messages. 

Tight  memory  and  communication  constraints  can  to  some  degree  be 
compensated  by  information  paging  techniques,  judicious  assignment  and 
collocation  of  entities,  judicious  handling  of  auxiliary  objects,  load  balancing, 
and  frequent  updates  of  the  global  system  time. 

4.6  General  Optimization  Stratesv 


Maximal  simulation  speed  can  be  obtained  through  an  iterative 
reorganization  of  the  simulation,  where  object  decompositions  are  varied. 
This  alternative  implies  changing  the  definition  of  simulation  objects  and  the 
assignment  of  both  objects  and  information  to  objects.  This  reassignment  and 
change  in  decomposition  strategy  would  modify  the  value  of  the  execution 
measure  components.  This  change  in  component  values  unfortunately  does  not 
have  a  straightforward  optimization  strategy  associated  with  it  because,  when 
one  component  is  decreased,  one  or  two  of  the  other  components  usually 
increase.  The  optimization  strategy  is  therefore  an  iterative  one  of  object 
redefinition,  and  reassignment  of  object  and  information. 
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5.0  FINDINGS  AND  OBSERVATIONS 
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Our  findings  and  observations  are  classified  in  three  categories: 
performance,  Time  VVarp/hypercube's  proposed  integration  with  object- 
oriented  programming,  and  projected  human  resource  requirements. 

5.1  Performance 

It  seems  clear  that  good  performance  on  the  Time  Warp/hypercube 
combination  requires  good  problem  decomposition  in  order  to  exploit 
parallelism.  Bad  decomposition  may  in  fact  produce  results  less  satisfactory 
than  straight  serial  processing. 

Secondly,  good  performance  will  depend  on  good  assignment  of  objects, 
behaviors,  and  data  among  processors.  This  assignment  is  necessary  to 
balance  the  load  between  processors  while  at  the  same  time  keeping  the 
processors  running  at  nearly  the  same  virtual  time.  It  is  desirable  to  minimize 
both  rollbacks  and  communications;  this  is  possible  in  part  by  load  balancing 
and  in  part  by  object  assignment. 

Because  there  are  no  clear  rules  of  thumb  to  determine  initial  good 
problem  decomposition  or  assignment  of  objects  to  processors,  testing  and 
iteration  will  be  required.  The  RAND  Corporation  is  currently  investigating 
techniques  and  possible  development  of  an  expert  system  for  determining 
object  assignment  to  take  advantage  of  parallel  processing. 

5.2  Integration  with  Object-Oriented  Programming 

The  use  of  the  Time  Warp/hypercube  mechanism  provides  a  favorable 
environment  for  the  use  of  object-oriented  programming  that  is  seen  as 
desirable  for  good  command  and  control  representation  in  a  simulation.  The 
environment  does  not,  however,  force  the  use  of  some  desirable  object- 
oriented  techniques,  e.g.,  not  all  of  the  necessary  support  features  are 
inherently  available  in  a  simulation  that  represents  objects,  but  does  not  have 
the  other  characteristics  of  object-oriented  simulation. 

The  object-oriented  design  of  the  hypercube  architecture  does  not  by 
itself  force  the  decomposition  of  the  problem  to  objects  making  a  good 


m 


representation  of  the  components  of  systems.  It  is  the  responsibility  of  the 
designer  to  cleverly  decompose  the  problem  into  objects  of  interest  for  the 
purpose  for  which  the  simulation  is  applied.  The  behaviors  must  also  be 
constructed  to  model  the  decisionmaking  processes  utilizing  information 
which  would  be  used  in  the  decision  process. 

In  the  Time  Warp/hypercube  mechanism,  message  passing  is  enforced 
when  the  objects  are  on  different  processors  and  also  when  they  are  on  the 
same  processor.  Because  of  the  message  passing,  the  ability  to  trace  cause 
and  effect  via  message  flow  is  inherently  capable  of  being  implemented. 

There  are  some  features  available  in  object-oriented  programming  which 
are  not  automatically  available  and  must  be  designed  into  a  support  package  if 
desired.  Parallel  processing  negates  much  of  the  utility  of  the  object 
hierarchy,  not  only  in  construction  of  the  simulation  but  also  in  the 
inheritance  of  object  properties  and  behaviors.  The  inheritance  of  properties 
from  generic  objects  is  made  difficult  when  generic  and  instance  objects  are 
not  collocated  on  the  same  processor;  this  is  because  the  instance  object  must 
query  the  generic  objects  for  its  inherited  behaviors  and  properties. 

5.3  Human  Resource  Requirements 

Because  of  the  problem  decomposition  complexity  and  the  need  for  good 
assignment  of  objects  of  processors,  it  appears  initially  that  there  will  be 
heavy  resource  (personnel,  effort,  and  time)  requirements  associated  with  the 
use  of  the  Time  Warp/hypercube  mechanism.  It  is  important  to  point  out  that 
such  a  cost  should  be  expected,  at  least  initially,  with  a  methodology  this 
new.  Because  there  is  no  large  base  of  experience  in  problem  decomposition 
and  object  assignment  problems  of  this  type,  considerable  experimentation 
and  trials  may  be  required  before  successful  results  are  attained. 

In  order  to  facilitate  the  use  of  the  Time  Warp/hypercube  mechanism, 
we  recommend  that  a  good  package  of  simulation  tools  be  made  available  to 
assist  the  user  in  the  construction,  use,  and  modification  of  the  object- 
oriented  simulations.  If  the  basic  language  of  the  object-oriented  simulation 
is  to  remain,  a  front  end  needs  to  be  provided  for  the  less  sophisticated  user. 
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6.0  CONCLUSIONS 


It  is  our  opinion  that  the  continued  funding  of  Time  Warp/hypercube  is 
worthwhile  and  that  it  will  contribute  towards  the  effort  to  obtain  improved 
representations  of  command  and  control  within  combat  simulations.  Although 
there  are  some  potential  restrictions,  we  do  not  see  any  clear  reason  why 
there  cannot  be  significant  gains  in  the  representation  provided. 

There  are  two  reasons  which  exist  for  limiting  the  results  to  be 
expected.  The  first  is  that  of  problem  decomposition  which  must  be 
accomplished  to  exploit  inherent  parallelism  while  at  the  same  time  attaining 
the  credibility  of  representation  and  the  goals  of  the  analysis  to  which  the 
simulation  is  to  be  applied.  The  second  reason  lies  in  the  trade-off  which 
must  be  made  between  the  amounts  of  processing  and  communications  due  to 
the  assignment  of  simulation  objects  to  processors. 

In  order  that  maximum  advantage  might  be  taken  of  the  Time 
Wsirp/hypercube  mechanism,  several  areas  of  research  need  to  be  pursued. 
The  first  is  how  to  "cleverly”  decompose  a  large  serial  simulation  for 
placement  on  a  parallel  architecture.  The  second  is  how  to  "smartly"  assign 
objects  to  the  processors  within  the  parallel  architecture.  The  third  area  is 
how  to  handle  databases  and  functions. 

In  the  near  term,  we  believe  that  implementing  a  command  and  control 
model  on  the  Time  Warp/hypercube  mechanism  will  require  considerable 
effort  for  the  reasons  previously  discussed. 
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Preltmlnory  Design  Concepts  on  Conmond  And 
Control  Modelling  Using  Time  Harp/Hypercube 


OUTLINE 


Objectives 
Problem  Background 
Previous  MITRE  Support 
Definitions 

Canmond  and  Control  Functions 

Implied  Required  Representotion  of  Objects. 

Doto.  Procedures 

•  Design  Alternotives 

•  Evaluation.  Observation  ond  Reconnendatlons 


objectives 


.  To  develop  prellmlnory  design  concepts  for  modelling 
on  Time  Woro/HypercuDe  (TW/HC) 

•  To  evaluate 

-  Performance 

-  Integration  with  current  architectures 

-  Effect  on  use  of  object  oriented  technloues 

-  Capability  to  deliver  better  C^  representation 
Resource  requirements 
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iUPPOKTXKC  MIA  BASIS 


SimUTlOiS 


PREVIOUS  HITRE  SUPPORT  (Cont'd) 


Use  Of  ROSS  In  Battlefield  Environment  Model 

Functlonol  Analysis  of  Comnand  and  Control  Process 
-  Moneover  control 


DEFINITIONS 


•  Object  oriented  Progromnlng 

•  Poroilel  Processing 

•  Hypercube 


OBJECT-ORIENTED  PROGRAWING 


Basic  Objects  represent  ports  of  system  being  studied 
Auxiliary  objects  sioy  be  used  for  computation  or  control 
Objects  hove  properties  ond  behaviors 
Hessoges  between  objects  invoke  behaviors 
Message  passing  olds  troclng  cause  ond  effect 


OBJECT  HIERARCHY 


Feature  of  some  object-oriented  progroinilng  languoges/systens 

Allows  storing  of  doto  ond  behaviors  ot  highest  common  level 
(reduces  storage) 

Aids  In  construction,  modificotlon  and  control  of  simulation 
Limited  use  on  hypercube 
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HYPERCUBE 


•  Parallel  processing 

•  2"  Processors 

•  n  Links 

•  (tax  n  'HOPS* 

•  All  processors  Identical 


SOnVARE  CONTROL 

TINE  MARP/VIRTUAL  TINE 

Allows  processors  to  run  asynchronously 

Reoulres  object-oriented,  message  passing  orchitecture 

Nessoges  Queues  and  stotes  periodically  saved 

Object  'rolls  bock*  to  a  previously-saved  state 
If  necessary 

Global  virtual  time  updated  perlodicolly 


TASKS 


•  Collection  Inforaotlon 

•  Evoluote  Plan<  coooOllltles.  reoutroients 

•  Oeveloo  Decision  (t  TronsRit) 

•  Allocote  Forces 

<  Coordinate 

•  Control  -  nonltor<  Adjust 


MITfS  EXAHPLE  ELEflENTS  OF  IHPUT  INFOHflATlOH 


COnPARlSON  OF  PERCEIVED  AND 
DESIRED  SITUATION 


Conranders  Guidance 
Friendly  Htsslon 
Friendly  Force  Status 
Constraints 
Enemy  Force  Stotus 
Key  Terroln 
Neother  Forecost 


OMi  Info 


outside  Info 


Decision  Crtterio 


RESOLUTION  OBJECTS 


•ACTION*  UNITS  -  BAHALIONS 


'OWN'  DATA 

-  Strengths  Location^  Status.  Superior.  Subordlnote.  Support. 
AOJocent.  Nlsston.  Plon 

DATA  BASES 

-  Terrain 

-  Heotner 

-  Enemy  Locations  (perceived) 

-  Apr  lor  1  knowledge  of  enemy,  tactics,  etc 

KNOWLEDGE  BASES 

-  Conditions  under  wttlch  to  report,  reouest.  support,  reouest 
relief,  break  contact,  etc. 


TOP  LEVEL  OBJECTS 


CONTROL  UNITS  •  CORPS  /  DIVISION  HEADQUARTERS 

'OWN  DATA'  -  HO.  aggregated,  ond  front  line  BNS 

-  Strength.  Location.  Status.  Superior.  Subordinate.  Support. 
Adjacent,  nisslon.  Plans 

DATA  BASES  (Present  and  Future) 

-  Terrain 

-  Heather 

-  Enemy  locations  (perceived,  out  to  72  hrs.  300  KRS) 

-  Present  ond  forecast  level  of  supplies  ond  replacements 

-  Aprlorl  knowledge  of  enemy,  tactics,  etc 

KNOWLEDGE  BASES 

-  Operation  plan  development  ond  dissemination 
Informotlon  receipt  and  processing 

-  Comparison  of  perceived  and  desired  sltuotlon 

-  Decision  to  modify/redo  plon 
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AUXILIARY  ACTORS 


•  Handle  concutottonol  load 

•  Hake  baste  actor  code  cleoner 

•  Localize  searches,  interaction 

•  Serve  os  Intemtedlory  'controller* 
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DESIGN  AND  EVALUATION  -  OUTLINE 


•  Design 

•  Evaluation 

•  Informotlon  ond  Procedure  Allocotlon  Strotegles 

•  Optimization  Strategy 

•  Auxiliary  Objects 

•  Implementation  Harnlnn  Slpnols 

•  Tightening  of  Assunptlons 
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DESIGN  CONSIDERATIONS 

simulation  objects  represent  military  units 

Other  objects  may  be  necessary  to  hondle  connon  procedures,  dotobose 

Functional  yorlonce  of  military  objects 

Hierarchical  relationship  of  objects 

Coupling  of  objects 

Amount  of  Interoctlon  between  objects 
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DESIGN  VARIABLES 

•  Allococlon  of  Informotlon  ond  behoylors  to  processors 

•  Allocation  of  objects  to  processors 
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ALLOCATION  OF  INFORNATION  AND  PROCEDURES  TO  OBJECTS 


•  Allocation  strategy  -  Object  gets: 

-  All  necessary  Infonratlon  ond  procedures 

-  some  Informotlon  ond  procedures 

-  Its  OMn  Informotlon 

•  Constraints 

-  Processor  memory  sUe 

-  Connunlcotlons  speed  ond  bondMldth 

-  Processor  speed 


ALLOCATION  OF  OBJECTS  TO  PROCESSORS 


•  Object  Definition 

•  Allocation  Strategies 

-  All  simulation  objects  on  one  processor 

-  One  slmulotlon  object  on  one  processor 

-  Several  slmulotlon  objects  on  one  processor  (clustering) 

•  Constraints 

-  Processor  memory  size 

-  Comnunlcdtlons  speed  ond  bondMldth 

-  Nunber  of  processors  ovolloble 

-  Processor  speed 

-  Nimber  of  messoges  passed  between  objects 
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Dwrees  of  freoOow  exist  In  lnplwentotlon 
HoM  to  evaluate  otternotlve  ollocotlon  strategies 
Evaluation  based  on  totol  execution  time 


EXECUTION  TWE 


Execution  time  is  defined  os  the  time  eloosed  from  Mhen  first 
processor  begins  to  when  the  lost  processor  finishes 

Execution  time  will  be  the  some  for  oil  processors 

E«P*C*S*W 

Where: 

P  •  Processlna  time 
C  •  Connunlcotlon  time 
S  •  Synchronlzotlon  time 
W  •  Idle  or  wait  time 

node!  assumes  one  serial  process  per  processor;  no  overlap  of 
processlna^  communication  and  synchronization  times 


DEFINITIONS 


•  Three  types  of  conmonicotlon: 

-  Ouery  Cg 

-  Uodote  C(j 

-  Action  Ca 


Two  causes  of  rollbock: 

-  Update  $1) 

-  Action  Sa 
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INFOnUTlON  AND  PROCEDURE  ALLOCATION  STRATEGIES 
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HESSAGE  FREQUENCY 

E 

Schema 

Nunter  of 
Queries 

Nmber  of 
updates 

P 

Cq 

c 

Ca 

Cu 

S 

h 

Su 

M 

•  All  necessory  in¬ 
formation  and 
procedures 

L 

H 

H 

L 

N/A 

H 

1 

1 

•  Some  Informotlon 
and  procedures 

n 

1 

n 

N/A 

n 

1 

1 

•  Own  information 

H 

B 

H 

N/A 

L 

H 

H  -  High 
H  •  Medlun 
L  •  Low 

N/A  •  No  Imooct 
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VARIANCES  IN  OBJECT  CHARACTERIZATION 

'S 

•  High  Level  Kllltory  Object/  e.g./  Corps  HO 

-  Plons 

-  Processes  Infomiotlon 

-  Issues  orders 

-  Receives  Informotlon 

•  Resolution  nilltory  Object,  e.g..  Battalion 

-  Gathers  Informotlon 

-  Receives  orders 

-  Sends  Informotlon 
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OPTIMIZATION  STRATEGY 

High  Level  Units 

E 

*  P  ♦  Cq  ♦  Ca  ♦  Cy  ♦  Sa 

high  ffledtum  H/A  high  ? 

♦  Sy  ♦  W 

?  medlun 

Probleiii  Areas 

High  P 

Solutions 

-  Allocate  object  to  more  than  one  processor  (lood  bolonce) 

High  Cy 

-  Decrease  onount  of  stored  Infonnotlon 

-  Decrease  In  Cy  Mill  increase  Cg  and  H 

?  Sa 

-  Minimize  local  virtual  time  difference 
receiver  of  action  by  lood  boloncing 

betMeen  sender  ond 

?Su 

-  Minimize  locol  vlrtuol  time  difference 
receiver  of  update  by  load  balancing 

between  sender  and 
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OPTIMIZATION  STRATEGY  (CONTINUED) 

Resolution  Object 

E  -  P  ♦  Cg  ♦  Ca  ♦  Cy  ♦  Sa  ♦  Sy  ♦  H 

low  hirh  N/A  low  ?  low  high 

Problems  Areos 

Solutions 

Low  P 

-  Cluster  low  level  objects  (lood  bolonce) 

High  Cg 

-  Increase  amount  of  Informotlon  In  memory 

-  Decreose  In  Cg  will  increose  Cy  ond  P  ond  decreose  H 

?  Sa 

-  Minimize  locol  vlrtuol  time  difference  between  sender 
and  receiver  of  octlons  by  lood  boloncing 

High  N 

1 

-  Increose  amount  of  Informotlon  In  memory 

-  Decrease  In  H  will  Increose  Cy 

\ _ J 
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AUXILIARY  OBJECTS 

•  Two  types 

-  Procedures 

-  Comnon  dotobose 

•  Potential  simulation  bottleneck 
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AUXILIARY  OBJECTS 


Procedures 

•  Not  time  based 

•  Provide  comoutattonol  functions,  e.p..  movement 

•  Decomposition  possible 

•  Allocation  Strotepy  •  Optlmtzotlon 

-  Replication 

-  Placement  with  objects 
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AUXILIARY  OBJECTS 

•  Tine  dependency 

•  Frequency  of  change  varies  greatly 

•  Freauent  change  can  cause  nojor  rollback  oroblens  os  mllltory 
objects  can't  discrimlnote  which  informtlon  Impacts  then 
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COItlON  DATABASE  AUXILIARY  OBJECTS 


For  databases  with  ifiw  freauency  of  chonge;  e.g.,  terroln 

-  Reollcate 

-  Pooe  to  objects 

-  nmimlze  local  vlrtuol  time  difference  between  dotobose 
ond  objects  which  use  It 

For  databases  with  hion  frequency  of  change;  e.g..  ground  truth 

•  Creote  Judicious  partitions 

•  Blnimlze  local  virtual  time  difference  between  oortltlons 
ond  those  objects  which  use  then 
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INPLENENTATION  WARNING  SIGNALS 

•  High  ntiiber  of  messages 

-  Bod  allocation  of  objects  to  processor 

•  High  nunber  of  ontl-messoges 

-  Bod  allocation  of  objects  to  processor 

-  Poor  clustering 

-  Poor  entity  definition 

-  Inherently  serial  problem 

•  High  nimber  of  rollbocks 

-  Bod  ol locations  of  objects  to  processor 

-  Processor  load  is  not  bolonced 

-  Inherently  serlol  problem 
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TIGHTENING  Of  ASSUNPTIONS 


CQII5TRAH1T5 


•  Processor  Nerory  Size 

-  Limits  the  nififeer  of  old  stote  copies  saved,  and  therefore  ease 
of  rollback 

-  Limits  degree  of  clustering 

•  Bounds  (mount  of  stored  Informotlon  ond  procedures 

•  SlOM  Conmunlcatlons 

•  SlOMS  simulation 

-  Increases  frequency  ond  mognitude  of  rollbocks 


•  Paging 

•  Judicious  allocation  ond  collocotlon  of  entitles 

•  Judicious  handling  of  ouxlllory  objects 

•  Load  balancing 

•  Freouenct  uodotes  of  Gvr 
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FINDINGS/OBSERVATIONS 


PERFORMANCE 


•  Perfomance  Mill  depend  on 

-  good  oroblen  deconposltlon 

good  object,  procedure  and  data  asslgiments  onong 
orocessars 

•  Testing  and  Iteration  required 


Major  bottlenecks 

seouentlol  nature  of  processes  at  higher  levels 
large  cannon  data  boses 
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INTEGRATION  WITH  CURRENT  ARCHITECTURES 


Hlerarchlcol  concept  Mould  be  focllltoted  If  FORCEM 
and  CASTFOREH  also  object-oriented 


(But)  flow  of  dota  (scenorlo.  results)  usuolly  requires 
Messaging  onyMoy 
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BEHER  REPRESENTATION 

Object  orientation  forces  designer /progromer  to 
think  through  process 

Object  orientation  provides  nodularity  Mhich  eases 
nodlflcotlon  of  represented 

HordMore/softtMre  nechonisn  to  provide  quick  enough 
processing  to  use  object-oriented  progrmilng  thereby 
obtaining  better  representation 
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RESOURCE  REOUIRENENTS 

Heavy  design  requirements 

Heavy  nodlflcotlon  rewilrements 

Dedicated  system  Mhen  operotlonal 

User  skill  level  dependent  on  noture  of  development 
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EFFECT  ON  USE  OF  OBJECT  ORIENTED  TECHNIQUES 

•  Design  gushes  probleni  Oeconoosltlon  to  objects 

•  Itessoge  passing  enforced 

•  Tracing  available 

•  Familiar  cooabllltles  not  ovolloble  In  prototype 
(and  C) 

•  Simulation  tool  package  reoulred 

-  hierarchy 
'  graphics 

-  modification 


CONCLUSIONS 


•  TH/HC  concept  worth  pursuing 

•  Two  reasons  to  limit  expectotlons 

-  difficulty  of  problem  decomposition 

-  trodeoffs  of  processing  and  communication 

•  To  take  maxlRun  odvontoge 

-  need  mnart  decomposition  of  problem 

-  need  smort  ossignment  of  objects 

need  aeort  woy  to  nondle  comnon  doto  bases 
and  functions 


Considerable  effort  In  construction  and  modification  of 
0  connand  and  control  model  linolemented  on  the  TH/HC 
mechanism  con  be  expected 

A  good  bockoge  of  slmulotlon  tools  is  needed 
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GLOSSARY 


AMIP 

AMMO 


COR  DIVE  M 


Army  Model  Improvement  Program 

Army  Model  Improvement  Program  Management  Office 

Battlefield  Environment  Model 

Command  and  Control 

Corps  and  Division  Evaluation  Model 
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